Papers with network architectures
Inspecting Unification of Encoding and Matching with Transformer: A Case Study of Machine Reading Comprehension (D19-58)
Copied to clipboard
| Challenge: | Experimental results show that unified model outperforms other models that treat encoding and matching separately. |
| Approach: | They evaluate a unified model with Transformer layers for machine reading comprehension . they find that the model learns different modeling strategies compared with previous models . |
| Outcome: | The unified model outperforms models with Transformer layers on the machine reading comprehension task. |
SURE: Mutually Visible Objects and Self-generated Candidate Labels For Relation Extraction (2025.coling-main)
Copied to clipboard
| Challenge: | Joint relation extraction models face high computational complexity, complex network architectures, difficult parameter tuning and limited interpretability. |
| Approach: | They develop a candidate label marker mechanism that prioritizes strategic label selection over simple label generation. |
| Outcome: | The proposed candidate label marks improve the SOTA methods by 2.5%, 1.9%, 1.2% . the proposed candidate labels improve the performance of the proposed methods . |
Improving Neural Language Models by Segmenting, Attending, and Predicting the Future (P19-1)
Copied to clipboard
| Challenge: | Common language models typically predict the next word given a past context. |
| Approach: | They propose a method that aligns the given context and the following phrase . they define syntactic heights and phrase segmentation rules to enable it to learn . |
| Outcome: | The proposed model outperforms strong baseline models on Wikitext-103 dataset. |
How Much Attention Do You Need? A Granular Analysis of Neural Machine Translation Architectures (P18-1)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) has been replaced by convolutional or self-attentional approaches. |
| Approach: | They propose an architecture definition language that allows for a flexible combination of common building blocks. |
| Outcome: | The proposed architectures can bring recurrent and convolutional models close to the Transformer architecture, but not using self-attention. |
Probing Linguistic Systematicity (2020.acl-main)
Copied to clipboard
| Challenge: | Existing evidence that deep natural language understanding models do not learn systematically is lacking. |
| Approach: | They examine whether deep natural language understanding models exhibit systematicity . they find that network architectures can generalize non-systematically . |
| Outcome: | The proposed model generalizes non-systematically, but is unsatisfactory, the authors argue . they show that the current state-of-the-art models do not generalize systematically . |
Double Path Networks for Sequence to Sequence Learning (C18-1)
Copied to clipboard
| Challenge: | Existing approaches for Sequence to Sequence learning have been developed . convolutional neural networks and self-attention networks are the most popular . |
| Approach: | They propose to integrate convolutional and self-attention layers into a double path network for sequence to sequence learning. |
| Outcome: | The proposed method significantly improves performance over state-of-the-art systems. |
Layer-Wise Multi-View Learning for Neural Machine Translation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to neural machine translation are limited to the topmost encoder layer’s context representation and cannot perceive the lower encoder layers. |
| Approach: | They propose a layer-wise multi-view learning approach to solve this problem by incorporating an auxiliary view into the model. |
| Outcome: | The proposed model can achieve stable results over multiple strong baselines and is agnostic to network architectures. |
NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Structured pruning methods have proven effective in reducing the model size and accelerating inference speed in various network architectures. |
| Approach: | They propose a framework that narrows the encoder and shortens the decoder networks of encoder-decoder models. |
| Outcome: | The proposed framework reduces the number of decoder layers and improves generation quality. |
Matching Distributions between Model and Data: Cross-domain Knowledge Distillation for Unsupervised Domain Adaptation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods require to learn to adapt the target model by exploiting the source data and sharing the network architecture across domains. |
| Approach: | They propose a framework that allows to transfer the knowledge of source domain to the unlabeled target domain without using source data. |
| Outcome: | The proposed framework matches distributions between a trained source model and a set of target data and achieves superior performance on cross-domain text classification. |
Defense against Synonym Substitution-based Adversarial Attacks via Dirichlet Neighborhood Ensemble (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show vulnerability of deep neural networks to adversarial examples that intentionally fool the networks. |
| Approach: | They propose a method for training a robust model to defense synonym substitution-based attacks by sampling embedding vectors for each word in an input sentence and augmenting them with the training data. |
| Outcome: | The proposed method outperforms other proposed defense methods by a significant margin across different network architectures and multiple data sets. |
Accelerating Transformer Inference for Translation via Parallel Decoding (2023.acl-long)
Copied to clipboard
Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, Emanuele Rodola
| Challenge: | Autoregressive decoding limits the efficiency of transformers for Machine Translation (MT) Existing methods to solve this problem are expensive and require changes to the model. |
| Approach: | They propose to reframe autoregressive decoding with a parallel formulation . they propose to speed up existing models without training or modifications while retaining translation quality. |
| Outcome: | The proposed model speeds up existing models without training or modifications while retaining translation quality. |
Exploiting the Syntax-Model Consistency for Neural Relation Extraction (2020.acl-main)
Copied to clipboard
| Challenge: | Existing deep learning models for Relation Extraction (RE) have limited generalization beyond the syntactic structures of the input sentences. |
| Approach: | They propose a deep learning model that uses dependency trees to extract syntactic importance of words for Relation Extraction. |
| Outcome: | The proposed model outperforms existing models on three RE benchmark datasets. |